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Background/context: 

Description of prior research, its intellectual context and its policy context. 



In the present education policy environment a high priority has been placed on improving 
teacher quality and teaching effectiveness in U.S. schools (Darling-Hammond et al., 2009; 
Obama, 2009). Standards-based educational improvement requires teachers to have deep 
knowledge of their subject and the pedagogy that is most effective for teaching the subject. 

States and school districts are charged with establishing and leading professional development 
programs, some with federal funding support, which will address major needs for improved 
preparation of teachers. The whole issue of teacher quality, including teacher preparation and 
ongoing professional development, and improving teacher effectiveness in classrooms, is at the 
heart of efforts to improve the quality and performance of our public schools. 

More recently, several major research synthesis projects have broadly analyzed evidence 
on the effects of mathematics and science teacher preparation and development initiatives on 
student achievement. One approach to reviewing evidence across studies is to apply a logic 
model and to examine the relationship of teacher preparation on student achievement through 
effects on intervening variables such as teacher knowledge and instructional practices (Clewell et 
al., 2004; Ingvarson, Meiers & Beavis, 2005). This kind of full analytic model allows educators 
and leaders to identify key decisions about the organization, delivery and support of teacher 
development that are ingredients to positive outcomes. 

State and local education agencies are responsible for directing and managing the use of 
federal funds for teacher development and improvement as well as guiding programs supported 
by states. Additionally, states are now required under NCLB to report on the qualifications of 
teachers in core academic subjects and the proportion of teachers that receive high quality 
professional development each year. Finally, states provide leadership for local systems on how 
to design, select, and implement professional development for teachers. Strong, research-based 
program designs, and evidence on their effects, are now in high demand across the U.S. 

States and in turn local districts seek models for designing and implementing effective 
professional development and particularly models supported by research evidence. 

The intended audiences for the study’s findings are education leaders, decision-makers 
and researchers. The study design builds on prior research and reporting on professional 
development programs and evaluation findings (Blank, de las Alas & Smith, 2007, 2008). The 
study was designed to measure and summarize consistent, systematic findings across multiple 
studies that show significant effects of teacher professional development on student achievement 
gains in K-12 mathematics or science. 

Purpose / objective / research question / focus of study: 

Description of what the research focused on and why. 



The meta analysis study focused on identifying and analyzing research studies that 
measured effects of teacher professional development with a content focus on math or science. 
The meta analysis was carried out to address two primary questions: 
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1) 



What are the effects of content-focused professional development for math and 
science teachers on improving student achievement as demonstrated across a 
range of studies? 

2) What characteristics of professional development programs (e.g., content focus, 
duration, coherence, active learning, and collective participation of teachers) 
explain the degree of effectiveness, and are the findings consistent with prior 
research on effective professional development? 

Setting: 

Description of where the research took place. 



The study took place in the United States over a period of two years from 2006 to 2008, 
with analysis extended to the first part of 2009. 

Population / Participants / Subjects: 

Description of participants in the study: who (or what ) how many, key features (or characteristics). 



Across all the studies reviewed, the focus was on teachers in public elementary and 
secondary schools teaching math or science at one or more grades K-12 and teachers who 
participated in a professional development program aimed to improve their teaching in math or 
science. 

Intervention / Program / Practice: 

Description of the intervention, program or practice, including details of administration and duration. 



The meta analysis identified 16 studies of programs that had significant effect sizes and 
provided teachers with professional development in mathematics or science. The information 
available on program interventions indicated they included combinations of learning activities 
such as summer institutes, coursework, study group, classroom mentoring, and professional 
networking. Eight of the programs also offered teachers opportunities to put into practice newly- 
learned lessons from the professional development by leading classroom instruction, and seven 
of the programs bring teachers to observe a classroom with either an exemplary teacher modeling 
instruction or a peer teacher implementing lessons learned during the professional development. 
More details about program characteristics are available in Table 2. dnsert Table 2 hero 

Research Design: 

Description of research design (e.g., qualitative case study, quasi-experimental design, secondary analysis, analytic 
essay, randomized field trial). 



Meta analysis 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 



The design for the meta analysis built on prior studies in education (Borman et al., 2002; 
Yoon et al., 2007; Lipsey & Wilson, 2001) and applied it to findings about professional 
development across states and districts. 

The design had four steps: 
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1) identification and collection of potential studies, 

2) determination of study eligibility and coding process, 

3) data analysis, 

4) reporting and dissemination. 

The design for the meta-analysis was also informed by a review of findings on teacher 
development programs conducted by the American Institute for Research (Yoon, et al., 2007). 
Figures 1 and 2 illustrates the process in more detail, dnsert Figures 1, 2>. In particular, the 
meta-analysis study design centered on two areas: capturing the characteristics of the 
professional development programs discussed in the studies and documenting the resulting 
measurable student outcomes the studies attribute to the professional development programs. 

The search process for potential studies included published and unpublished works as 
well as evaluation reports from funded state and federal professional development projects. The 
study authors conducted an intensive electronic search, using multiple and well-known databases 
and meta-databases. In addition, searches were conducted targeting certain periodicals in which 
evaluation studies of professional development programs would be featured. Publications and 
databases of major education research centers were also examined. Moreover, the study authors 
contacted principal investigators listed by program grants from the U.S. Department of 
Education Title II-B project evaluations and the research studies funded by the Institute of 
Education Sciences, the NSF Teacher Preparation Continuum and MSP project evaluations, and 
studies of the Local Systemic Initiatives. Lastly, cross-checks were carried out with findings 
from prior reviews and synthesis studies in teacher professional development. Four hundred 
sixteen studies or reports were identified for pre-screening. A review of the corresponding 
abstracts of those studies reduced the count to 74 studies. These remaining studies were 
screened by a team of trained coders who utilized a coding form based on a coding and 
reconciliation software program and form developed by AIR (Yoon, et al., 2007). Figure xx 
outlines the document review process and the resulting studies included in the meta analysis, 
dnsert Figure xx>. Meta regression analyses were conducted with the remaining sixteen 
studies, with the focus on studies that featured professional development in mathematics, since 
these studies produces the greater number of effect sizes than professional development in 
science. 

Findings / Results: 

Description of main findings with specific details. 



The meta analysis of studies of teacher professional development programs in 
mathematics and science found that 16 studies reported significant effect sizes for teacher 
development in relation to improving student achievement. The evidence for the findings in the 
16 studies was based on scientific research designs. These studies reported effect sizes for 
student achievement gains for a treatment group as compared to a control group and the studies 
provided adequate data and documentation for the research team to compute or re-analyze effect 
sizes. The large majority (12 of 16) studies were focused on analyzing mathematics teacher 
professional development and effects on student achievement in mathematics. The mean effect 
size for mathematics studies using a pre-post design is 0.21. These results show consistent 
positive effect on gains in student achievement in mathematics from teacher professional 
development in mathematics education. Table 1 details mean effect size findings for math 
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studies by research design and measure type, dnsert Table 1>. Effect sizes were larger when 
measures of achievement were used that were specifically selected or developed to be aligned 
with the content focus of the professional development. However, the review of research did 
identify several studies with significant effects using large-scale statewide assessment programs. 

Several common patterns were found across the sixteen studies on professional 
development program designs. The program designs included strong emphasis on teachers 
learning specific subject content as well as pedagogical content for how to teach the content to 
students. The implementation of professional development included multiple activities to 
provide follow-up reinforcement of learning, assistance with implementation, and support for 
teachers from mentors and colleagues in their schools. In terms of duration of development 
activities, 14 of 16 programs that were reported continued for six months or more. The mean 
contact time with teachers in program activities was 91 hours. 

The numbers of teachers that were involved in the programs that were analyzed and 
found to be effective varied from less than ten to more than 90. The research and evaluation for 
the 16 studies employed multiple measures of student achievement and outcomes. The studies’ 
analysis of effects on student achievement included scales to measure learning in specific content 
areas (e.g., algebra, measurement). The use of multiple measures allowed use of different types 
of test items. A majority of the studies analyzed professional development for elementary and 
middle grades teachers. The analysis of effects showed a pattern of stronger effects for 
elementary level professional development than for middle or high school teachers. Table xx 
details the professional development program features found in the studies. 

The results for the 16 studies with effect sizes demonstrates to the education research and 
policy communities how meta analysis can and should be used in education to provide 
comparisons and aggregations of research findings over time and across many different studies. 
The process of review and analysis employed involved several thousand citations, initial pre- 
screening of 400-plus documents, and intensive coding and review of 74 studies. The methods 
of identifying, coding, and quantifying data used in the study can be employed for a variety of 
objectives in education research. 

Conclusions: 

Description of conclusions and recommendations based on findings and overall study. 

Based on the results of the meta analysis of findings from teacher professional 
development studies, several recommendations can be made about use of meta analysis methods 
and their use for researchers, evaluators and education leaders. 

• The meta-analysis design and procedures employed in the study proved to be 
effective in identifying a set of common findings regarding effects of teacher 
professional development on student achievement, and the procedures proved 
useful to determine which studies and their results met high standards for 
scientific validity and reliability. 

• A scientific research design can be efficiently employed to evaluate teacher 
professional development, and a design to measure effects of teacher development 
on subsequent student achievement should be strongly considered for each funded 
program for teacher and teaching improvement. 

• The use of research designs involving treatment and control groups should 
become a regular practice and built into the plan and organization for professional 
development and other initiatives. 
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• Measures of implementation of professional development are critical to 
evaluation design in order to document and measure activities to reinforce and 
extend learning for teachers in their school setting. 

• Multiple measures of student achievement should be included in the research 
design if possible to provide for different types of assessments of learning and 
analysis of subject content learned. 

• State and local education leaders should ensure that data systems are structured so 
that data on teacher development initiatives can be linked to student achievement 
measures, and these data can be effective for evaluation even where individual 
identifiers are removed. 

• Procedures for meta analysis modeled in this study provide a consistent, 
quantified methodology for application and use in other studies, including initial 
identification, multiple coding and validation of reviews, comparison of research 
design with established criteria, and consistent procedures for effect size analysis 
and coding of treatment variables. 

This meta analysis review did not include systematic identification or review of 
intervening measures of the professional development treatment, such as measures of gains in 
teacher knowledge, improvement in practices, or fidelity of implementation of what was learned. 
Several of the studies identified did report analysis of differences on these kinds of measures 
between teachers in the treatment and control groups. Further analysis across studies would 
provide stronger evidence and useful information about the relationship between professional 
learning of teachers from a specific initiative and subsequent improved learning by students. 
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Figure 1 : Logic Model 
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Figure 2 Steps in Study Design 




Table 1: Identified Studies and Key Characteristics & Effect Sizes* 









Treatment 
Teachers N 


Median 








Study 


Grade/ School Level; 


Size (All 


Effect 


Number of 




Study 


Design 


Content Area 


Teachers) 


Size 


Effects 


Student Outcome Measure 


Carpenter, et 
al., 1989 


RCT 


Elementary; Math 


20 (40) 


.39 


7 


ITBS (Level 7) 










.68 




Oral test 










.32 




Study-specific tests (Scale 1 ,2,3) 


Dickson, 


QED 


Gr. 8-10; Science 


4(8) 


.1 
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Texas Assessment of Academic Skills 


2002 
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Heller et al., 
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48 
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Math Pathways and Pitfalls pre-posttest 
gain 


Jagielski, 
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Gr. 3-8; Math 


43 (70) 


.77 


20 


MCIP/89 using released NAEP pre- 


1991 












posttest gain 


Lane, 2003 
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Elementary; Math 


1 2 (22) 


.13 
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Constructed CSAP pre-posttest gain 


META 
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Gr. 6-8; Math 


1 9 (34) 


.13 
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Colorado Student Assessment Program 


Assoc., 2006 
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Colorado Student Assessment Program 


Assoc., 2007 
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Meyer & 
Sutton, 2006 
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10 


Metropolitan Achievement Test posttest 
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Criterion Referenced Test posttest 


Niess, 2005 
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Gr. 3-8; Math 


24 (42) 


.11 


4 


Technology Enhanced State Assessment 
pre-posttest gain 



For Cohen's d an ES > 0.0 < 0.3 is a "small" effect, >0.3 and <0.8 a "medium" effect and >0.8 a "large" effect. 





Table 1 - continued 









Treatment 
Teachers N 


Median 








Study 


Grade/ School Level; 


Size (All 


Effect 


Number of 




Study 


Design 


Content Area 


Teachers) 


Size 


Effects 


Student Outcome Measure 


Palmer & 


QED 


Gr. 5-10; Science 


1 6 (43) 


.11 


5 


Northwest Evaluation Association 


Nelson, 

2006* 












assessments pre-posttest gain 


Rubin & 


RCT 


Middle; Science 


7(16) 


.64 


8 


Middle Grades Integrated Process Skill 


Norman, 

1992 












Test pre-posttest 










.12 




Group Assessment of Logical Thinking 
Test pre-posttest 


Saxe, 

Gearhart, & 
Nasir, 2001 


QED 


Elementary; Math 


17(6) 


XX 




Study-specific assessments 
(Computational Scale) 










1.63 


6 


Study-specific assessments (Conceptual 
Scale) posttest 


Scott, 2005 


QED 


Gr. 3; Science 


3(6) 


.20 


2 


Iowa Test of Basic Skills pre-posttest gain 


Siegle & 

McCoach, 

2007 


RCT 


Gr. 5; Math 


7(15) 


.20 


2 


Math Achievement Test 


Snippe, 1 992 


RCT 


High; Math 


87 (198) 


-.01 


21 


Terra Nova 










.20 




ACCUPLACER 










.06 




WorkKeys 


Walsh- 


QED 


Gr. 5; Math 


4(6) 


.26 


2 


PSG Achievement Assessment pre- 


Cavazos, 

1994 












posttest gain 




Table 2: Professional Development Features 



Study 

Authors, Year 


Professional Development 


Location 


Contact 

Hrs. 


Duration 
(in mo.) 


PD 

Components 


Carpenter, et al., 1 989 


Cognitively Guided 
Instruction 


24 schools in Madison, 
Wl metropolitan area 


80 


4.5 


Coursework 
Mentoring, Network 
Study group 
Summer institute 


Dickson, 2002 


Inquiry Institute Science 


Suburban school 
district north central 
Texas 


24 


8 


Internship 

Networking 


Heller et al., 2007 


Mathematics Pathways and 
Pitfalls 


Five diverse districts 
across the U.S. 


10 


8 


Internship 
Leading instruction 
Summer institute 


Jagielski, 1 991 


Mathematics Curriculum 
Improvement Project 


Chicago, IL 


36 


8 


Conference 
Leading instruction 
Networking 
Study group 


Lane, 2003 


Problem-solving and 
reasoning Math 


Five schools from one 
district in Colorado 


17 


8 


Classroom observation 
Developing assessment 
Study group 


META Assoc., 2006; 
2007 


Northeast Front Range 
Math/Science Partnership 


Five school districts in 
Colorado front range 


120 


7.5 


Coaching 

Devel. assessment 
Leading instruction 
Mentoring, Network 
Summer institute 


Meyer & Sutton, 2006 


Math in the Middle Institute 
Partnership 


Lincoln, NE 


540 


16 


Coursework 
Summer institute 





Table 2 - continued 



Study 

Authors, Year 


Professional 

Development 


Teachers Location 


PD Provider 
Agency 


Contact 

Hrs. 


Duration 


PD 

Components 


Niess, 2005 


High Desert MSP 
Math teaching 


Five school districts in 
central Oregon 


Oregon State 
University 


304 


8 


Classroom observation 
Leading instruction 
Networking 
Summer institute 


Palmer & 
Nelson, 2006 


REC Lesson Study 
Science 


Ten school districts in 
Minnesota 


University, 

Global 

Resources 


60 


8 mos. 


Classroom observation 
Developing assessment 
Networking 
Leading instruction 
Study group 
Summer institute 


Rubin & 
Norman, 1992 


Systematic 
Modeling Strategy 
Science Teaching 


Detroit, Ml 


Wayne State 
University 


30 


3 mos. 


Courses 

Mentoring 


Saxe, 

Gearhart, & 
Nasir, 2001 


Integrating 

Mathematics 

Assessment 


Los Angeles 
metropolitan area 


Researchers/ 

Authors 


41 


8 mos. 


Classroom observation 
Developing assessment 
Internship, Study 
groupLeading instruction 
Mentoring, Network Summer 
institute 





Table 2 - continued 



Study 

Authors, Year 


Professional 

Development 


Teachers Location 


PD Provider 
Agency 


Contact 

Hrs. 


Duration 


PD 

Components 


Scott, 2005 


TEAMS Professional 
Development Model 


Suburban-Urban 
district Texas 
metropolitan area 


School District 


168 


8 mos. 


Classroom observation 
Coaching 
Conference 
Leading discussion 
Mentoring 
Networking 
Study group 
Summer institute 


Siegle & 
McCoach, 2007 


Self-Efficacy 
Teaching Strategies 
& Implementation 
Math 


Ten districts varying 
urban, suburban, 
rural in six states 
(MA, MD, Ml, MT, 
NC, NE) 


University of 
Connecticut 


2 


1 day 


Coaching 

Leading instruction 
Networking 


Snippe, 1 992 


National Research 
Center for Career/ 
Tech Education 


Teachers from 
several states; 
providers traveled to 
each location 


University of 
Minnesota 


14 


3 days 


Classroom mentoring 
Networking 
Study group 


Walsh- 

Cavazos, 1994 


Probability, Statistics, 
and Graphing Module 


South Texas school 
district 


Researcher/ 

Author 


12 


3 days 


In-service activity 








Mean 

Range 


91 hrs. 

2 - 540 hrs. 


6 mos. 

1 day-16 
months 


3.3 activities 
1 - 6 activities 





Table 3: Mean Effect Sizes for Teacher Professional Development Effects On Student Achievement, Mathematics Studies [include only part a math] 



Categories 


Pre-Post Effect Size 
(SE) 


N Effects 


Post-Only 
Effect Size (SE) 


N Effects 


Math Studies 


0.21 ( 0 . 08 ) 


21 


0.13 ( 0 . 03 ) 


68 


Research Design 


RCT 


0.27 ( 0 . 13 ) 


5 


0.26 ( 0 . 05 ) 


35 


QED 


0.17 ( 0 . 08 ) 


16 


0.04 ( 0 . 04 ) 


33 


Measure Tvoe 


PD Specific 


0.32 ( 0 . 08 ) 


15 


0.28 ( 0 . 09 ) 


25 


State Criterion- Referenced 


0.01 ( 0 . 08 ) 


6 


- 0.07 ( 0 . 14 ) 


7 


National Norm-Referenced 


- 


- 


0.17 ( 0 . 04 ) 


25 


Local Test 


- 


- 


0.05 ( 0 . 02 ) 


11 



N Effects = number of effect sizes per category (across studies identified with at least one significant effect size); details on statistical tests in Appendix. 

[ 




Table 4: Correlation Table of Math Post-Only Professional Development Design Elements 





1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


13 


Time 

1 . Contact Hr. 


1 


























2. Frequency 


.741** 


1 
























3. Duration 


.834** 


.623** 


1 






















PD Activities 

4. Summer 
Institutes 


.577** 


.399** 


.655** 


1 




















5. College 
Courses 


.744** 


-.171 


.596** 


.618** 


1 


















6. Conferences 


-.196 


.094 


.146 


-.403** 


-.249* 


1 
















7. Study Group 


-.694** 


-.253 


-.602** 


-.524** 


-.369** 


.287* 


1 














Active Learning 

8. Lead Discussion 


-.196 


.094 


.146 


-.403** 


-.249* 


1.000** 


.287* 


1 












9. Learning Network 


-.657** 


.048 


-.601** 


-.351** 


-.471** 


.249* 


.796** 


.249* 


1 










10. Develop Assessments 


-.138 


.398** 


.135 


.345** 


-.249* 


-.172 


.021 


-.172 


.155 


1 








1 1 . Observe Teachers 


-.154 


.562* 


.084 


.418** 


-.360** 


-.249* 


-.298* 


-.249* 


-.093 


.692** 


1 






12. Classroom Mentoring 


-.421** 


-.571** 


-.742** 


-.394** 


-.028 


-.347** 


.579** 


-.347** 


.502** 


-.347** 


-.502** 


1 




Coherence 

13. Link to curriculum, 
Goals 


.043 


-.161 


.106 


-.406** 


-.244* 


.221 


.163 


.221 


-.158 


-.080 


-.324** 


-.059 


1 



Two-tail test: * significant at p<.05; ** significant at pc.01 
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Appendix: Detailed significance tests for Table 3 



Pre-Post Post Only 



Categories 


95 % Cl 


Q statistic 


95 % Cl 


Q statistic 


Math Studies 


(0.06, 0.36) 


Or = 153.72* 


(0.07, 0.20) 


Or = 328.78* 


Research Design 




Qb(1) =46.12* 




Qb(1) =66.72* 


RCT 


(0.01, 0.53) 


Qw = 53.24* 


(0.16, 0.35) 


Qw = 78.37* 


QED 


(0.01, 0.34) 


Qw = 54.35* 


(-0.04, 0.11) 


Qw= 183.70* 


Measure Tvoe 




Qb(1) = 84.46 




Qb(3) = 90.43* 


PD Specific 


(0.16, 0.49) 


Qw = 46.81 


(0.10, 0.46) 


Qw = 91 .73* 


State Criterion- Referenced 


(-0.15, 0.16) 


Qw = 22.45 


(-0.35, 0.21) 


Qw= 111.25* 


National Norm-Referenced 




— 


(0.10, 0.24) 


Qw = 16.33 


Local Test 




— 


(0.02, 0.09) 


Qw= 19.05* 



Notes: *p < .05; if Or is significant a random-effects model is applied. See further If Qw is not significant a fixed-effects model is applied. If Gw is significant a random- 
effect model is used for that category. Q B refers to differences between groups. 
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